Method, system, and program for recovery from a failure in an asynchronous data copying system

ABSTRACT

A method of recovery from a data storage system failure in a data storage system having a host computer writing data updates to a local storage controller at a local site. The local controller is associated with a local storage device. The local storage controller is also configured to asynchronously copy the updates to a remote storage controller associated with a remote storage device at a remote site. In addition, the remote storage controller is configured to store a consistent point in time copy of the updates on a backup storage device. The consistent point in time copy is known as a consistency group. Upon detection of a failure associated with the local site, a determination is made whether a group of updates pending for storage on the backup storage device form an intact consistency group. If an intact consistency group has not formed, corrective action may be taken to create an intact consistency group. The recovery method further consists of synchronizing the remote storage device, initiating recovery operations and, upon recovery of the local site, resynchronization of the local storage device and the backup storage device to recovery consistency group without the need for full volume storage copies and while minimizing application downtime.

RELATED APPLICATIONS

This application incorporates by reference commonly-assigned andco-pending U.S. patent Ser. No. 10/464,024, filed Jun. 6, 2003, andentitled METHOD, SYSTEM AND ARTICLE OF MANUFACTURE FOR REMOTE COPYING OFDATA. This application also incorporates by reference commonly-assignedand co-pending Ser. No. ______ Docket Nos. TUC9-2003-0116US1, entitledMETHOD, SYSTEM AND ARTICLE OF MANUFACTURE FOR RECOVERY FROM A FAILURE INA CASCADING PPRC SYSTEM; Ser. No. ______ TUC9-2003-0045US1, entitledAPPARATUS AND METHOD TO COORDINATE MULTIPLE DATA STORAGE AND RETREIVALSTORAGE, SYSTEMS; Ser. No. ______ TUC9-2003-0108US1, entitled METHOD,SYSTEM AND PROGRAM FOR FORMING A CONSISTENCY GROUP; Ser. No. ______TUC9-2003-0109US1 entitled AUTONOMIC INFRASTRUCTURE ENABLEMENT FOR POINTIN TIME COPY CONSISTENCY GROUPS; Ser. No. ______ TUC-2003-0100US1,entitled METHOD, SYSTEM, AND PROGRAM FOR MIRRORING DATA AMONG STORAGESITES; and Ser. No. ______ TUC9-2003-0119US1, entitled METHOD, SYSTEMAND PROGRAM FOR ASYNCHRONOUS COPY, all filed on Sep. 29, 2003.

TECHNICAL FIELD

The present invention relates to a method, system and article ofmanufacture for recovery from a failure associated with a storage sitein an asynchronous data copying system.

BACKGROUND ART

Disaster recovery systems typically address two types of failures: asudden catastrophic failure at a single point in time, or data loss overa period of time. In both types of failure scenario, updates to volumesmay be lost. To assist in recovery of data updates, a copy of data maybe provided at a remote location. Such dual or shadow copies aretypically made as the application system is writing new data to aprimary storage device. International Business Machines Corporation(IBM), the assignee of the subject patent application, provides thefollowing systems for maintaining remote copies of data at a secondarysite, Extended Remote Copy (XRC) and Peer-to-Peer Remote Copy (PPRC).These systems provide a method for the continuous mirroring of data to aremote site to failover to during a failure at the primary site fromwhich the data is being continuously mirrored. Such data mirroringsystems can also provide an additional remote copy for non-recoverypurposes such as local access at a remote site. These IBM XRC and PPRCsystems are described in the IBM publication “Remote Copy:Administrator's Guide and Reference,” IBM document number SC35-0169-02(IBM Copyright 1994, 1996), which publication is incorporated herein byreference in its entirety.

In such backup systems, data is maintained in volume pairs. A volumepair is comprised of a volume in a primary storage device and acorresponding volume in a secondary storage device that includes anidentical copy of the data maintained in the primary volume. Typically,the primary volume of the pair will be maintained in a primary directaccess storage device (DASD) and the secondary volume of the pair ismaintained in a secondary DASD shadowing data from the primary DASD. Aprimary storage controller may be provided to control access to theprimary DASD and a secondary storage controller may be provided tocontrol access to the secondary DASD. In the IBM XRC environment, theapplication system writing data to the primary volumes includes asysplex timer which provides a time-of-day (TOD) value as a time stampto data writes. The host system time stamps data sets when writing suchdata sets to volumes in the primary DASD. The integrity of data updatesis related to insuring that updates are done at the secondary volumes inthe volume pair in the same order as they were done on the primaryvolume. In XRC and other prior art systems, the cross systems commontime stamp provided by the system on behalf of the application programdetermines and maintains the logical sequence of data updates across anynumber of data volumes on any number of storage systems. In manyapplication programs, such as database systems, certain writes cannotoccur unless a previous write occurred; otherwise the data integritywould be jeopardized. Such a data write whose integrity is dependent onthe occurrence of a previous data write is known as a dependent write.For instance, if a customer opens an account, deposits $400.00, and thenwithdraws $300.00, the withdrawal update to the system is dependent onthe occurrence of the other writes, i.e., the opening of the account andthe deposit. When such dependent transactions are copied from theprimary volumes to the secondary volumes, the transaction order must bemaintained to maintain the integrity of the dependent write operation.

Volumes in the primary and secondary DASDs are consistent when allwrites have been transferred in their logical order, i.e., all dependentwrites transferred first before the writes dependent thereon. In thebanking example, this means that the deposit is written to the secondaryvolume before the withdrawal. A consistency group is a collection ofupdates to the primary volumes such that dependent writes are secured ina consistent manner. For instance, in the banking example, this meansthat the withdrawal transaction is in the same consistency group as thedeposit or in a later group; the withdrawal cannot be in an earlierconsistency group. Consistency groups maintain data consistency acrossvolumes and storage devices. For instance, if a failure occurs, thedeposit will be written to the secondary volume before the withdrawal.Thus, when data is recovered from the secondary volumes, the recovereddata will be consistent.

A consistency time is a time the system derives from the applicationsystem's time stamp to the data set. A consistency group has aconsistency time for all data writes in a consistency group having atime stamp equal or earlier than the consistency time stamp. In the IBMXRC environment, the consistency time is the latest time to which thesystem guarantees that updates to the second volume are consistent. Aslong as the application program is writing data to the primary volume,the consistency time increases. However, if update activity ceases, thenthe consistency time does not change as there are no data sets with timestamps to provide a time reference for further consistency groups. Ifall the records in the consistency group are written to secondaryvolumes, then the reported consistency time reflects the latest timestamp of all records in the consistency group. Methods for maintainingthe sequential consistency of data writes and forming consistency groupsto maintain sequential consistency in the transfer of data between aprimary DASD and secondary DASD are described in U.S. Pat. Nos.5,615,329 and 5,504,861, which are assigned to IBM, the assignee of thesubject patent application, and which are incorporated herein byreference in their entirety.

One technique to maintain consistency across copies is to time stampdata across primary volumes using a common clock source, referred to asa sysplex timer. Updates will be transferred in groups defined as allupdates having a time stamp less than a certain time. When clocksynchronization cannot be easily implemented to form consistency groupsacross systems, then another technique for forming consistency groups isto determine a cut off point. Any updates to primary volumes managed bythe primary controller cache dated as of the cut off point aretransferred to the secondary controller for storage in the secondaryvolumes. While transferring the data in the consistency group, theprimary storage controller would return busy to any host request whilethe data in the consistency group is transferred. After the data in theconsistency group is transferred and the primary and secondary storagecontroller are synchronized, i.e., any updates prior to the cut offpoint are transferred, then the primary controller would cease returningbusy to the applications. This ensures that the primary and secondaryvolumes are consistent as of the freeze cut off point.

As is discussed above and as is fully discussed in the commonly assignedand simultaneously filed U.S. application entitled “METHOD, SYSTEM, ANDPROGRAM FOR FORMING A CONSISTENCY GROUP”, Ser. No. ______ attorneydocket number TUC920030108US1, a storage system failure can result froma sudden or catastrophic failure at a single point in time. Such afailure can be particularly disruptive if a storage controller orstorage volume at a primary or local site fails since host I/Ooperations typically write to the storage system at the local site.

Currently known asynchronous data copying solutions have severalscenarios where the resumption of normal operations after a failure atthe primary or local site requires the customer to perform a full copyof all volumes maintained at a secondary or recovery site. Full volumecopies can take many hours depending on the amount of data stored in therespective volumes. Furthermore, full volume copies can leave thecustomer exposed to subsequent failures until normal operations can beresumed.

In addition, a data storage system configured across multiple storagesites and having multiple storage volumes and controllers may rely onconsistency group formation and consistency group processing to maintaindata consistency across volumes and storage devices. A need exists inthe art for a mechanism to facilitate the maintenance and manipulationof consistency groups across multiple storage controllers when failurestrikes a local controller associated with a local or primary site.Proper use of consistency groups can assure that recovery from a localfailure will proceed with minimal data loss and without the need for atime consuming full volume copy.

The present invention is directed toward overcoming one or more of theproblems discussed above.

SUMMARY OF THE INVENTION

The need in the art is addressed by a method of recovery from a datastorage system failure in a data storage system having a host computerwriting data updates to a local storage controller at a local site. Thelocal controller is associated with a local storage device. The localstorage controller is also configured to asynchronously copy the updatesto a remote storage controller associated with a remote storage deviceat a remote site. In addition, the remote storage controller isconfigured to store a consistent point in time copy of the updates on abackup storage device. The consistent point in time copy is known as aconsistency group.

The recovery method is triggered by the detection of a failureassociated with the local site. Upon detection of a failure associatedwith the local site, a determination is made whether a group of updatespending for storage on the backup storage device form an intactconsistency group. The recovery method ends with the correction of thefailure associated with the local site and resynchronization of thelocal storage device.

After detection of the failure associated with the local site, therecovery method may consist of terminating the asynchronous copying ofupdates from the local storage controller to the remote storagecontroller and establishment of a reverse asynchronous update copyingrelationship from the remote storage controller to the local storagecontroller. Furthermore, host computer I/O operations are typicallyquiesced upon detection of the failure associated with the local site.

If it was determined initially that a group of updates pending forstorage on the backup storage device did not form an intact consistencygroup, corrective action may be taken to create an intact consistencygroup. Depending on the status of the formation of the pendingconsistency group, the corrective action may consist of completing thepending consistency group, withdrawing the pending consistency group,reverting to a prior, intact consistency group, or allowing the pendingconsistency group to become the intact consistency group.

The recovery method may also consist of physically copying to the backupstorage device, the updates of the intact consistency group forming arecovery consistency group. The method may further consist ofsynchronizing the remote storage device to the recovery consistencygroup.

During the time period when the local site is experiencing a failure,I/O updates can be written directly to the remote storage controller.The post failure updates may be written to the remote storage controllerfrom a remote or recovery host.

Upon correction of the failure associated with the local site, the localstorage device must be synchronized with the most current data stored onthe system, which can comprise synchronizing the local storage devicewith the recovery consistency group and the post failure updates.Resynchronization of the local storage device can occur over the reverseasynchronous data copying relationship established from the remotestorage controller to the local storage controller.

After the local storage device is resynchronized, the recovery processcan be completed by quiescing the recovery host, and terminating theasynchronous copying of updates from the remote storage controller tothe local storage controller. In addition, the asynchronous updatecopying relationship from the local storage controller to the remotestorage controller can be reestablished. Finally, a new consistent copyof the data on the remote storage device may be stored on the backupstorage device. The new consistent copy is preferably made by mergingpost failure updates written to the remote storage device with therecovery consistency group existing on the backup storage device. Thus,synchronization of both the local storage device and the backup storagedevice can be accomplished without the need for a full volume copy. Inaddition, host application downtime is minimized.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a network computing environmentin which aspects of the invention are implemented;

FIG. 2 illustrates an arrangement of a storage controller in accordancewith implementations of the present invention;

FIG. 3 illustrates relationship information in accordance withimplementations of the present invention;

FIG. 4 illustrates component within the storage controllers of FIG. 1 inwhich aspects of the invention are implemented;

FIGS. 5-9 illustrate operations performed by storage controllers to copydata to a remote site in accordance with the present invention;

FIG. 10 illustrates an architecture of computing components in thenetwork environments, such as the storage controllers, used withimplementations of the present invention; and

FIG. 11 illustrates operation performed by the storage controllers torecover from a local site failure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 1. Detailed Descriptionof Underlying Data Copy System

In the following description, reference is made to the accompanyingdrawings which form a part hereof and which illustrates severalembodiments of the present invention. It is understood that otherembodiments may be utilized and structural and operational changes maybe made without departing from the scope of the present invention.

FIG. 1 illustrates a network computing environment in which aspects ofthe invention may be implemented. A local site 2 includes storagecontrollers 4A, 4B, 4C where each storage controller 4A, 4B, 4C iscoupled to an attached storage system 6A, 6B, 6C, respectively. Thestorage controllers 4A, 4B, 4C manage input/output (I/O) access tovolumes in the storage systems 6A, 6B, 6C from local host 18. Eachstorage controller 4A, 4B, 4C copies data to a corresponding remotestorage controller 8A, 8B, 8C at a remote site 10 via connections 12A,12B, 12C. The storage controllers 4A, 4B, 4C transmit data consistent asof a point of time to the remote storage controllers 8A, 8B, 8C, wherethe remote storage controllers 8A, 8B, 8C store the data to theirrespective attached remote storage systems 14A, 14B, 14C. In certainimplementations, the storage controllers 4A, 4B, 4C continuouslytransmit data to the remote site 10, and at certain instances in timewill ensure the consistency of the data at the remote site 10. Incertain implementations, the remote site 10 may save a consistency groupof data to another storage, such as consistent backup storages 16A, 16B,16C to maintain a backup of the consistent data at the remote site 10.After a consistent group of data is transmitted to the remote storage14A, 14B, 14C, a copy is made from remote storage 14A, 14B, 14C to arespective consistent backup storage 16A, 16B, 16C. Although FIG. 1illustrates a certain number of components at the sites, such as threestorage controllers and storages, the described implementations mayoperate with any number of storage controllers and storage systems atthe sites.

The remote site 10 may be hundreds or thousands of miles from the localsite 2. The connections 12A, 12B, 12C between pairs of storagecontrollers may comprise a dedicated line, such as a dedicated fiberoptical network or connection maintained by a long distancetelecommunication carrier, long distant fiber optic connection havingchannel extenders, or extend through a network such as a storage areanetwork (SAN), local area network (LAN), the internet, an intranet, etc.

Local host 18 at the local site 2 may perform I/O operations withrespect to volumes in the storage systems 6A, 6B, 6C. The local host 18may communicate with storage controllers 4A, 4B, 4C via any network orconnection known in the art. The storage controllers 4A, 4B, 4C at thelocal site may have certain master-slave relationships with respect tothe volumes managed by the storage controllers. For instance, storagecontroller may be designated as a master, e.g., 4C, and other designatedas slaves, e.g., 4A, 4B, where certain volumes managed by the designatedslaves are managed by the storage controller designated as the master.In this way, one master storage controller may manage remote copyoperations for specified volumes for slave controllers designated asslaves for a particular master.

The storage controllers 4A, 4B, 4C and remote storage controllers 8A,8B, 8C further include a processor complex and may comprise any storagecontroller or server known in the art such as the IBM Enterprise StorageServer (ESS)®, 3990 Storage Controller, etc. (Enterprise Storage Server®is a registered trademark of IBM). The storage systems 6A, 6B, 6C,remote storage systems 14A, 14B, 14C and backup storage systems 16A,16B, 16C may comprise an array of storage devices such as just a bunchof disks (JBOD), redundant array of independent disks (RAID),virtualization device, etc.

FIG. 2 illustrates programs and information that may be included in astorage controller 4 at the local site 2, e.g., storage controllers 4A,4B, 4C, including storage management software 62, relationshipassignment information 64, an out of synch (OOS) bitmap 66 and a changerecording (CR) bitmap 68. The storage management software 62 may includecode to allow for remote copying of data such as the code included inthe IBM PPRC Extended Distance program to allow for mirroring of dataover relatively short and long distances. Further details of the IBMPPRC Extended Distance program are described in the IBM publication “IBMTotalStorage Enterprise Storage Server PPRC Extended Distance,” IBMdocument number SG24-6568-00 (Copyright IBM, June 2002), whichpublication is incorporated herein by reference in its entirety. Therelationship information 64 maintained by the storage managementsoftware 62 includes information for each volume (or portion of avolume) on the storage systems 6A, 6B, 6C subject to a consistency grouprelationship. For master-slave environments such as that shown in FIG.1, the relationship information 64 further indicates whether the storagecontroller is a master for a relationship or whether a volume in theattached storage is a slave in a point in time copy relationship. Astorage controller designated as the master storage controller, e.g.,4C, manages consistency group copy operations for designated volumes inits attached storage, e.g., 6C, or volumes in other storage systems,e.g., 6A, 6B, managed by other storage controllers, e.g., 4A, 4B,designated as slaves. Thus, the master-slave designation may be made ona volume by volume basis. In alternative implementations, themaster-slave designation may be made at a sub-volume level. Differentvolumes in one storage controller may be assigned as slaves to differentpoint in time copy relationships. Thus, a master storage controller maycontrol the point in time copy for volumes distributed in the storagesystems managed by different storage controllers.

The local storage controllers 4A, 4B, 4C may communicated over a localnetwork 18 which may comprise direct connections or a switched network.Alternatively, the storage controllers 4A, 4B, 4C may communicate over alink 12A, 12B, 12C if the links also extend through a network at thelocal site 2.

In further implementations, the mirroring of data and consistency groupmanagement may occur on a session basis. A session may be defined asincluding multiple volumes that may be referenced using a session namewhen the session is started. A master-slave relationship may be assignedto a session so that relationship would apply to all volumes in thesession. All volumes defined within a session would be included in thesame consistency group.

The relationship information 64 may include a plurality of relationshipassignments, one for each volume for which the storage controller isdesignated as a master or slave. FIG. 3 illustrates information includedin a relationship assignment 80. A relationship identification field 82indicates an ID of a consistency group copy relationship because theremay be many relationships, each assigned to a group of volumes, acrossthe storage controllers 4A, 4B, 4C. Thus, a storage controller 4A, 4B,4C may be a master for multiple consistency group relationships or havevolumes that are slaves in different relationships. The master-slavefield 84 indicates whether the assignment is for a master or slaverelationship. If the master-slave indicator 84 specifies “master”, thenthat storage controller, e.g., 4C, including that assignment 80, is amaster in the master-slave relationship identified in field 82 for thevolume indicated in the volume information 88 managed by the storagecontroller, e.g., 4A or 4B, indicated in the storage controllerinformation 86. Likewise, if the master-slave indicator 82 specifies“slave”, then the volume in the attached storage, e.g., 6A or 6B,indicated in the volume information 88 is a slave in the relationshipidentified in field 82 to the master storage controller, e.g., 6 b,indicated in the storage controller information 86. A master storagecontroller, e.g., 4C, may be assigned a slave volume that is in storage,e.g., 6A, 6B, attached to another storage controller, e.g., 4A, 4B, oris in the storage, e.g., 6C, attached to the master storage controller,e.g., 4C. Thus, a master may be assigned slave volumes in the attachedstorage and/or in storage systems attached to other storage controllers.Further, a storage controller may be assigned as a master for volumes inthe attached storage and/or storages managed by other storagecontrollers and, at the same time, include volumes that are assigned asslaves to master, where that master may be the storage controller itselfor another storage controller.

Further, the assignment information 64 for a master storage controllerwould include a relationship record 80 for each slave volume on a slavestorage controller, where the slave storage controller and volume wouldbe indicated in fields 86 and 88, respectively. The relationshipinformation may be maintained in a different arrangement and format thanshown in FIG. 3 and may be maintained in any data structure known in theart, such as a text or formatted text file, database, etc.

An OOS bitmap 66 includes a bit for each track that the slave storagecontrollers are to copy over to the corresponding remote storage. Duringnormal copy operations, any writes to a storage controller are recordedin the OOS bitmap 66 and then copied over to the remote storage. The CRbitmap 68 includes a bit for each track to which an update is receivedduring the formation of a consistent set of data, where the formationoccurs by the master storage controller, e.g., 4C, managing the slaves,e.g., 4A, 4B, to copy all the data reflected in their OOS bitmaps 66 asof a point-in time. Further, the storage controllers 4A, 4B, 4C maymaintain one set of these bitmaps for each volume, or alternativelymaintain one set of bitmaps 66, 68 for a group of volumes in theattached storage 6A, 6B, 6C.

In certain implementations, the storage controllers always run abackground process to copy any tracks indicated as updated in their OOSbitmaps to the remote storage. This process continues in the slavecontrollers even after the formation of a consistency group is initiatedand the CR bitmap created. FIG. 4 illustrates further details of thestorage controllers 4A, 4B, 4C where each is referred to as a primarystorage controller 102 that copies data to a secondary storagecontroller 104, such as storage controllers 14A, 14B, 14C which receivethe data. The primary storage controller 102 includes a primary cache106 in which updates to tracks in the attached primary storage 108managed by the primary storage controller 102 such as storages 6A, 6B,6C are maintained until written to the attached primary storage 108(i.e., the tracks are destaged). Additionally, the primary storagecontroller 102 includes a nonvolatile cache 110. The nonvolatile cache110 may be, for example, a battery backed up volatile memory to maintaina nonvolatile copy of data updates and other information.

The primary storage controller 102 includes a read process 112 forreading data and a write process 114 for writing data. The primarystorage controller 102 also includes an async process 116 fortransferring data from the primary storage controller 102 to remotestorage such as storage at the secondary storage controller 104. Incertain implementations, the async process 116 may performsynchronization of data for an asynchronous PPRC Extended Distanceservice. In certain implementations, there may be multiple asyncprocesses (e.g., one async process for each volume on primary storage108 for PPRC Extended Distance). In certain implementations, the readprocess 112, write process 114, and async process 116 are implemented asfirmware. The secondary storage controller 104 writes the data receivedfrom the primary storage controller 102 to secondary storage 120 such asstorages 14A, 14B, 14C (FIG. 1). The different processes 112, 114, 116may be implemented in storage management software 62.

FIG. 5 illustrates operations performed by a master storage controller,e.g., 4C, to manage a point in time copy operation. A master storagecontroller may implement a policy that indicates when the master willinitiate the formation of a consistency group to copy the volumessubject to the consistency group relationship. Additionally, the mastermay initiate consistency group formation operations in response to anadministrator command. Upon the master storage controller initiating (atblock 100) the formation of a consistency group with respect to aspecified consistency group relationship, the master storage controller,e.g., 4C, queries (at block 102) each slave storage controller, e.g.,4A, 4B, in the specified relationship to determine whether that slavehas completed any remote copy operations for the slave volume subject tothe relationship. As discussed, the master storage controller, e.g., 4C,for the specified relationship would include a relationship record 80(FIG. 3) for each slave volume in the specified relationship identifyingthe slave storage controller in field 86 and slave volume 88, where theslave storage controller may be the master or a different storagecontroller. Upon determining (at block 124) that all slave controllershave completed remote copying for their respective slave volumes, then acommand is transmitted to each slave storage controller, e.g., 4A, 4B,to create a new CR bitmap 68 for the slave volume in the specifiedrelationship. The master storage controller may learn that the slaveshave completed all remote copying by querying the slaves or by theslaves notifying the master when remote copying is complete.

FIG. 6 illustrates operations the slave storage controllers, e.g., 4A,4B, perform in response to receiving the command to generate the CRbitmap 68 for the slave volume. Upon receiving (at block 150) thecommand to generate the CR bitmap 68 for the slave volume, the slavestorage controller, e.g., 4A, 4B, queues (at block 152) any subsequentwrites directed to the slave volume without returning complete. The CRbitmap 68 for the slave volume is generated (at block 154) and a messageis sent (at block 155) to the master indicating the initialization ofthe CR bitmap 68. Upon receiving (at block 157) a message or commandfrom the master storage controller, e.g., 4C, to proceed with writes,queued writes to the slave volume are allowed to complete (at block 156)and the bit for the updated track in the CR bitmap 68 is set toindicated the update. Alternatively, indication of the queued writes maybe made to the OOS bitmap 66 instead of the CR bitmap 68. In this way,the formation of the consistency group only delays host writes for ashort period of time while the CR bitmap 68 is generated, where suchdelay during which host writes are queued may not even be noticeable ormay appear to the host as a slightly longer write. The slave storagecontroller, e.g., 4A, 4B, then completes any subsequent writes to theslave volume (at block 158) and indicates the update to the track in theCR bitmap 68 for the slave volume in storage, e.g., 6A, 6B. The slavestorage controller, e.g., 4A, 4B, further signals (at block 160) themaster storage controller, e.g., 4C, of completing the CR bitmap 68.

FIG. 5 also illustrates the master storage controller, e.g., 4C, receipt(at block 126) of complete messages from all the storage controllers,e.g., 4A, 4B, indicating that they have completed generating the CRbitmap 68 for their slave volumes and then sends (at block 127) a runmessage to the slave storage controllers to signal them to proceed withtheir writes.

FIG. 7 illustrates operations the slave storage controllers, e.g., 6A,6B perform when copying the tracks OOS bitmap 66, which occurs as partof an asynchronous background process that continually runs. The slavestorage controller asynchronously copies (at block 182) each trackindicated in the OOS bitmap 66 for the slave volume to the correspondingremote storage controller, e.g., 8A, 8B, and sets the bit for thecorresponding track in the OOS bitmap 66 to “off” when completing theasynchronous copy. The slave storage controller signals (at block 184)the master storage controller when all tracks indicated in the OOSbitmap 66 for the slave volume have been asynchronously copied to thecorresponding remote storage controller at the remote site 10 if the OOSbitmap 66 was involved in the formation of a consistency group. If theOOS bitmap 66 did not represent volumes included in a consistency groupin the process of being formed, then there may be no notification to themaster of the completion of the copying of the tracks for that OOSbitmap 66.

Further, storage controllers asynchronously copying the data indicatedin their OOS bitmaps 66 to the remote storage may perform conflictmanagement operations. For instance, if a storage controller 4A, 4B, 4Creceives an update to a track whose corresponding bit in the OOS bitmap66 is set to indicate that there already is an update for that trackthat has not yet been copied over to the remote site, where such olderupdate may or may not be part of a consistency group, then the storagecontroller 4A, 4B, 4C may take action to avoid overwriting the olderupdate until such update has been copied over. Such conflict managementwould ensure an older update that is part of a consistency group iscopied to the remote site to ensure consistency at the time of formationof the consistency group. In certain implementations, the storagecontroller receiving the update may queue the update until the olderupdate is copied over to the remote site. Alternatively, the storagecontroller may return a “fail” to the update while there is a pendingupdate to copy over. Still further, the storage controller may make acopy of the older update, write the most recent update to the track, andthen later copy over the older update from the copy made. Additionaldetails of managing conflicts with updated tracks indicated in the OOSbitmap are describe in the copending and commonly assigned patentapplication entitled “METHOD, SYSTEM, AND PROGRAM FOR ASYNCHRONOUSCOPY”, Ser. No. ______ having attorney docket number TUC920030119US1,which patent application is incorporated herein by reference in itsentirety.

With respect to FIG. 5, when the master storage controller, e.g., 4C,determines (at block 108) that all slave storage controllers, e.g., 4A,4B, have completed copying their slave volume to their respective remotestorage controller, e.g., 8A, 8B, the master storage controller sends acommand to each slave controller to cause the slave controller to, inturn, send a command to cause the respective remote storage controller8A, 8B to make a copy of the slave volume stored in the remote storage,e.g., 14A, 14B, to the consistent backup storage, e.g., 16A, 16B. Incertain implementations, the copy operation from the remote storage 14A,14B, 14C to the consistent backup 16A, 16B, 16C may comprise a virtualcopy operation, such as a FlashCopy® operation (FlashCopy is aregistered trademark of IBM). A FlashCopy® operation involvesestablishing a logical point in time relationship between source andtarget volumes on the same or different devices and subsequently copyingthe data subject to the relationship from the source to target volumes.The FlashCopy® operation guarantees that until a track in a FlashCopy®relationship has been hardened to its location on the target disk, thetrack resides on the source disk.

FIG. 8 illustrates operations the slave storage controllers perform inresponse to receiving (at block 200) the command to cause the virtualcopy operations between the remote storage, e.g., 14A, 14B, and theconsistent backup, e.g., 16A, 16B. In response to the command, the slavestorage controller, e.g., 4A, 4B, sends (at block 202) a command to thecorresponding remote storage controller, e.g., 8A, 8B, to cause theremote storage controller to make a virtual copy of the slave volume tothe corresponding consistent backup storage, e.g., 16A, 16B. The slavestorage controller then signals (at block 204) the master storagecontroller of the result of the remote virtual copy operation.

With respect to FIG. 5, the master storage controller determines (atblock 130) whether the remote virtual copy command sent by every slavestorage controller has successfully completed. This may be determined bythe master querying the slaves or by the slaves notifying the masterwhen learning the result of the remote virtual copy operation. If (atblock 130) all virtual copy operations initiated by all slaves havecompleted successfully, then an “increment complete” message is sent (atblock 132) to every slave storage controller to complete the point intime copy operation and the point in time copy operation ends with“complete” (at block 134). Otherwise, if the virtual copy operation hasnot completed successfully for all slaves, then the master storagecontroller sends (at block 136) a command to each slave storagecontroller whose remote virtual copy operation succeeded to reverse thevirtual copy operation and return to previous point in time copy on theconsistent backup, e.g., 16A, 16B. Further details of reversing thevirtual copy operation are described in co-pending and commonly assignedpatent application entitled “AUTONOMIC INFRASTRUCTURE ENABLEMENT FORPOINT IN TIME COPY CONSISTENCY GROUPS”, Ser. No. ______ having attorneydocket number TUC920030109US1, which application is incorporated hereinby reference in its entirety.

FIG. 9 illustrates operations the slave storage controllers perform inresponse to receiving (at block 220) the “increment complete” messagefrom the master storage controller. In response, the slave storagecontrollers merge (at block 222) the set bits of the CR bitmap 68 forthe slave volume, indicating updated tracks, into the drained OOS bitmap66 and subsequent writes to the slave volume are indicated in the OOSbitmap 66, not the CR bitmap 68 which may now be discarded.

With the described implementations, volumes subjected to a consistencygroup relationship may reside on storage systems managed by differentstorage controllers. A master storage controller would coordinate theremote copying of data at the different storage controllers when formingthe consistency group.

The OOS bitmap 66 and CR bitmap 68 may be used in alternative remotecopy environments. For instance, in a single controller environment,without any master-slave relationships, the storage controller mayreceive a consistency group formation command to copy specified volumesmanaged by the storage controller to a remote site. In response, thestorage controller would queue any further writes while generating theCR bitmap 68. After generating the CR bitmap 68, any queued writes andsubsequent writes would be indicated in the CR bitmap 68 and tracksindicated in the OOS bitmap 66 would be copied to the remote site. Afterthe OOS bitmap 66 is drained and the asynchronous remote copy operationsare completed, new writes may again be recorded in the OOS bitmap 66,the CR bitmap 68 is merged with the OOS bitmap 66, and then the CRbitmap 68 may be discarded. Further, a virtual copy may be performed atthe remote site of the volumes, consistent as of the time of the storagecontroller receiving the point in time copy command.

The described techniques to form consistency groups may be implementedas a method, apparatus or article of manufacture using standardprogramming and/or engineering techniques to produce software, firmware,hardware, or any combination thereof. The term “article of manufacture”as used herein refers to code or logic implemented in hardware logic(e.g., an integrated circuit chip, Programmable Gate Array (PGA),Application Specific Integrated Circuit (ASIC), etc.) or a computerreadable medium (e.g., magnetic storage medium such as hard disk drives,floppy disks, tape), optical storage (e.g., CD-ROMs, optical disks,etc.), volatile and non-volatile memory devices (e.g., EEPROMs, ROMs,PROMs, RAMs, DRAMs, SRAMs, firmware, programmable logic, etc.). Code inthe computer readable medium is accessed and executed by a processor.The code in which implementations are made may further be accessiblethrough a transmission media or from a file server over a network. Insuch cases, the article of manufacture in which the code is implementedmay comprise a transmission media such as network transmission line,wireless transmission media, signals propagating through space, radiowaves, infrared signals, etc. Of course, those skilled in the art willrecognize that many modifications may be made to this configurationwithout departing from the scope of the implementations and that thearticle of manufacture may comprise any information bearing medium knownin the art.

II. Detailed Description of Method for Recovery from Local Site Failure

A generalized illustration of a method for recovery from a failureassociated with a local site 2 is shown in FIG. 11. As discussed indetail above, the local site 2 may have storage controllers, e.g., 4A,4B, 4C, connected by a local network 18. In addition, each storagecontroller 4A, 4B, 4C may have an associated storage device 6A, 6B, 6C.The failure associated with the local site 2 may be a failure of any oneor more of these components, which make up the local site. As isdescribed in detail above and as is shown in FIG. 1, each local storagecontroller 4A, 4B, 4C communicates over a connection 12A, 12B, 12C to aremote storage controller 8A, 8B, 8C. The remote storage controllers areassociated with remote storage devices 14A, 14B, 14C and consistentbackup storage devices 16A, 16B, 16C. To enhance the simplicity andclarity of the discussion below, the present invention of a method forrecovery from a local site failure shall be limited to a discussion of alocal site 2 having a single local storage controller 4 associated witha local storage device 6. Similarly, for the purposes of the discussionbelow, the description of the remote site 10 shall be limited to aremote site 10 having a single remote storage controller 8 associatedwith a remote storage device 14 and a backup storage device 16. Thisconvention employed to simplify the discussion of the recovery systemand method should in no way be construed as limiting upon the scope ofthe invention which is wholly applicable to a system for copying datawhich features multiple master and slave storage controllers andmultiple storage devices configured across multiple sites.

When the failure associated with the local site 2 occurs, the balance ofthe data storage system is initially unaware of the failure. Highavailability cluster multiprocessing (HACMP), geographically dispersedparallel sysplex (GDPS), or other management software detects thefailure associated with the local site 2 (step 510). Upon detection ofthe failure associated with the local site 2, asynchronous update copyoperations from the local storage controller 4 to the remote storagecontroller 8 must be terminated (step 512). In addition, once thefailure associated with the local site 2 has occurred, the storagemanagement software 62 associated with a local storage controller 4 willno longer be able to form consistency groups which include the localstorage device 6. Therefore, it is necessary to query the remote storagecontroller 8 to determine the status of virtual copy operations pendingat the time of the local site failure (step 514), and thereby determinethe status of any consistency group which may have been in formation atthe time of the failure.

In addition to querying the remote storage controller 8 to determine thestatus of pending consistency group formation, corrective action can betaken on any pending consistency group to create an intact consistencygroup as a basis for further recovery operations (step 516). Thecorrective action may consist of completing the pending consistencygroup. Alternatively, the corrective action may consist of withdrawingthe pending consistency group and/or allowing reversion to a priorintact consistency group. Finally, the corrective action may consist oftaking no action and allowing the pending consistency group to becomethe intact consistency group. Preferably, a tape or other type ofsecondary backup of the backup storage device 16 should be made afterthe status of the consistency group formation is checked and correctiveaction is taken. The tape or other secondary backup ensures a safe copyis preserved before proceeding with the recovery process.

At this point in time, the pending intact consistency group is a logicalconsistency group, meaning that it exists on the remote storagecontroller 8 but has not yet been physically copied to the backupstorage device 16. Before the recovery process can continue, the pendingintact consistency group must be hardened (physically copied to) on thebackup storage device 16 (step 520) creating a recovery consistencygroup.

In addition to the preliminary steps described above, an asynchronousupdate copy relationship from the remote storage controller 8 to thelocal storage controller 4 must be established prior to activation ofthe recovery host 20 (step 518). The reverse asynchronous update copyrelationship may be established by the issuance of a failover command tothe remote storage controller 8 establishing a remote to local copypath. Preferably, all failover operations must be completed beforecontinuing with the next step. In addition, this command suspends therelationship between the remote storage controller 8 and the localstorage controller 4 and causes changes to the remote storage device 14to be reflected in an OOS bitmap 66.

Next, change recording must be enabled on the backup storage device 16to record post failure updates processed by the remote storagecontroller 8 and thus avoid a full volume copy from the remote storagedevice 14 to the backup storage device 16 when the originalconfiguration is restored (step 522). In addition, a reverse point intime copy relationship which can be a reverse FlashCopy relationship maybe initiated between the backup storage device 16 and the remote storagedevice 14 to assure that the remote storage device 14 is operating withthe recovery consistency group recovered from the backup storage device16 (step 524). The reverse point in time (FlashCopy) operation causesthe remote storage device 14 to become a target for the backup storagedevice 16. Therefore, the CR bitmap 68 of the remote storage device 14(which represents changes on tracks of the remote storage device 14received from the local storage device 6 that are not in the consistentcopy) will be merged into both the target bitmap 69 and the OOS bitmap66 of the remote storage device 14 and subsequently zeroed. Thus, theremote storage device 14 becomes a copy of the backup storage device 16(the consistent copy). These mapped updates also represent changed dataon the local storage device 6. Therefore, these updates are marked inthe remote storage device 14 OOS bitmap 66 such that the updates will bereset on the local storage device 6 in later steps. In addition, thebackup storage device CR bitmap 68 will be reflecting recovery host 20activity on the backup storage device 16 once the recovery host 20 isbrought up in the next step.

Upon completion of the above preliminary steps and operations, therecovery host 20 can be activated. The customer will be using theconsistency group recovered from the backup storage device 16. Therecovery host 20 preferably will run the customer's application andallow customer operations during the time that the local site failure isbeing repaired. The recovery host I/O operations can be configured towrite data which is processed by the remote storage controller 8 (step526) and stored. Between the time that the recovery host I/O operationsbegin writing through the remote storage controller 8 and the time thatthe failure associated with the local site 2 is repaired, the identityof tracks containing post failure data updates on the remote storagedevice 14 are identified with the OOS bitmap 66 and CR bitmap 68associated with the remote storage controller 8.

Upon recovery from the local site failure, and prior to the resumptionof normal operations, the local storage device 6 must be resynchronizedwith the remote storage device 14 (step 528). When the local storagedevices 6 is ready to return, the OOS bitmap 66 of the local storagecontroller 4 can be merged with the OOS bitmap 66 of the remote storagecontroller 8 to assure that updates written to either controller priorto or after the failure are accounted for as part of theresynchronization process.

In particular, resynchronization of the local storage device 6 may beaccomplished through the following steps which will allow forresynchronization using existing storage system functions. A failbackcommand can be issued to the remote storage controller 8. This commandcauses the OOS bitmap 66 from the local storage controller 4 to bemerged with the OOS bitmap 66 of the remote storage controller 8 beforethe remote to local volume pairs are reestablished. When the merger iscomplete, the resynchronization from the remote storage device 14 to thelocal storage device 6 will be started. Resynchronization occurs byeither the asynchronous or the synchronous copying of changed datastored on tracks identified by the OOS bitmap 66 of the remote storagecontroller 8 to the local storage device 6.

Once the remote storage controller 8 has completed a firstresynchronization pass through its OOS bitmap 66, recovery host 20operations (which are still directed at the backup storage device 16)may be quiesced. Subsequently, the remaining bits can be allowed todrain from the OOS bitmap 66, causing the OOS bitmap 66 to read allzeroes. At this time, the CR bitmap 68 associated with the remotestorage controller 8 (presently all zeroes) will continue recordingchanges written in the following steps. Next, the remote to local volumepairs can be reversed to the pre-failure configuration again by issuinga failover command and then a failback command to the local storagedevice 6. New updates are still recorded in the CR bitmap associatedwith the remote storage controller 8. Subsequently, a reverse FLCrestore command (FlashCopy restore) can be issued to reverse thedirection of the FlashCopy relationship to the pre-failure configurationand to establish a new consistency group on the backup storage device16. As part of this process, the updates to the remote storage device 14(recorded in the CR bitmap 68) will be merged with the backup storagedevice target bitmap 69 so the FlashCopy function is signaled to copythe updated tracks. At this point in time, the local storage device 6 isa synchronized copy of the remote storage device 14, which is asynchronized copy of the backup storage device 16. Thereupon, primaryhost operations may be resumed (step 538). Finally, the pre-failureasynchronous PPRC mechanism can be reinitiated. Since change recordinghad been maintained during the recovery process to track changes madefrom the point in time that the recovery consistency group was hardened,the establishment of a new consistency group on the backup storagedevice 16 can be accomplished without the need for a full volume copy.If the recovery process becomes corrupted, the recovery consistencygroup may still be available if a tape copy was made as discussed above.

The above implementations were described with respect to the IBM PPRCExtended Distance computing environment. However, the describedimplementations for maintaining consistency during recovery processescould be applied to maintain consistency in other computing and vendorenvironments and using other data copying protocols and programs thandescribed herein.

In certain implementations, data in the storage devices is arranged involumes. In alternative systems, the storage areas on which data ismaintained may be grouped according to storage units other than volumesthat are grouped in groupings other than sessions for the purpose ofmaintaining consistency.

In described implementations, the storage units designated as the slavesin the master-slave copy relationship comprised volumes. In alternativeimplementations, other storage unit values may be assigned as a slave toa master where such storage units may comprise tracks in a volume,blocks, logical subsystems, logical drives, or any other physical orlogical storage unit designation known in the art.

In described implementations, data updates were organized in consistencygroups before being transferred from one storage controller to another.In alternative implementations, the data copied between the differentstorage controllers may not comprise a consistency group.

The described operations for forming a consistency group may be used ina critical data environment where high availability is paramount.However, those skilled in the art will appreciate that the point in timecopy operations described herein may apply to storage systems used fornon-critical data where high availability is not absolutely necessary.

In the described implementations, each storage controller copies data toa corresponding remote storage controller having an attached remotestorage system. In alternative implementations, multiple storagecontrollers at the local or intermediate site may copy data to oneremote storage controller where a single remote storage controller mayreceive point in time data from multiple local or intermediate storagecontrollers.

The illustrated logic of FIGS. 5-9 and FIG. 11 show certain eventsoccurring in a certain order. In alternative implementations, certainoperations may be performed in a different order, modified, or removed.Moreover, steps may be added to the above described logic and stillconform to the described implementations. Further, operations describedherein may occur sequentially or certain operations may be processed inparallel. Yet further, operations may be performed by a singleprocessing unit or by distributed processing units.

FIG. 10 illustrates one implementation of a computer architecture 400 ofthe network components such as the storage controllers shown in FIGS. 1and 9. The architecture 400 may include a processor 402 (e.g., amicroprocessor), a memory 404 (e.g., a volatile memory device), andstorage 406 (e.g., a non-volatile storage such as magnetic disk drives,optical disk drives, a tape drive, etc.). The storage 406 may comprisean internal storage device or an attached network accessible storage.Programs in the storage 406 are loaded into the memory 404 and executedby the processor 402 in a manner known in the art. The architecturefurther includes a network card 408 to enable communication with anetwork. An input device 410 is used to provide user input to theprocessor 402 and may include a keyboard, mouse, pen-stylus, microphone,touch sensitive display screen, or any other activation or inputmechanism known in the art. An output device 412 is capable of renderinginformation transmitted from the processor 402 or other component suchas a display monitor, printer, storage, etc.

The foregoing description of various implementations of the inventionhas been presented for the purposes of illustration and description. Itis not intended to be exhaustive, nor to limit the invention to theprecise form disclosed. Many modifications and variations are possiblein light of the above teaching. It is intended that the scope of theinvention be limited not by this detailed description, but rather by theclaims appended hereto. The above specification, examples and dataprovide a complete description of the manufacture and use of thecomposition of the invention. Since many embodiments of the inventioncan be made without departing from the spirit and scope of theinvention, the invention resides in the claims hereinafter appended.

The objects of the invention have been fully realized through theembodiments disclosed herein. Those skilled in the art will appreciatethat the various aspects of the invention may be achieved throughdifferent embodiments without departing from the essential function ofthe invention. The particular embodiments are illustrative and not meantto limit the scope of the invention as set forth in the followingclaims.

1. A method of recovery from a data storage system failure in a datastorage system having a host computer writing updates to a local storagecontroller associated with a local storage device at a local site, thelocal storage controller asynchronously copying the updates to a remotestorage controller at a remote site the remote storage controllerstoring the updates on a remote storage device and periodically storinga consistent point in time copy of the updates on a backup storagedevice, the method comprising: detecting a failure associated with thelocal site; determining whether a consistent point in time copy of theupdates pending for storage on the backup storage device at the time thefailure is detected form an intact consistency group; correcting thefailure associated with the local site; and re-synchronizing the localstorage device.
 2. The method of claim 1 further comprising thefollowing steps upon detecting the failure associated with the localsite: terminating the asynchronous copying of updates from the localstorage controller to the remote storage controller; establishing areverse asynchronous update copying relationship from the remote storagecontroller to the local storage controller; and quiescing the hostcomputer.
 3. The method of claim 1 further comprising taking correctiveaction on the updates pending for storage on the backup device to createan intact consistency group.
 4. The method of claim 3 wherein thecorrective action taken is selected from a group of actions consistingof completing a pending consistency group, withdrawing the pendingconsistency group, reverting to a prior intact consistency group andallowing the pending consistency group to become the intact consistencygroup.
 5. The method of claim 4 further comprising physically copying tothe backup storage device the updates in the intact consistency group,thereby forming a recovery consistency group.
 6. The method of claim 5further comprising synchronizing the updates stored on the remotestorage device with the updates of the recovery consistency group. 7.The method of claim 6 further comprising writing post-failure updatesdirectly to the remote storage controller.
 8. The method of claim 7wherein the post-failure updates are written to the remote storagecontroller from a remote host.
 9. The method of claim 7 whereinre-synchronizing the local storage device comprises synchronizing thelocal storage device with the recovery consistency group and thepost-failure updates by the asynchronous copying of updates from therecovery storage controller to the local recovery controller.
 10. Themethod of claim 7 further comprising the following steps after the localstorage device is re-synchronized: quiescing the recovery host;terminating the asynchronous copying of updates from the remote storagecontroller to the local storage controller; re-establishing theasynchronous copy relationship from the local storage controller to theremote storage controller; and storing a new consistent copy of the dataon the remote storage device to the backup storage device.
 11. Themethod of claim 10 wherein storing a new consistent copy of the data onthe remote storage device to the backup storage device comprises mergingpost-failure updates written to the remote storage volume with therecovery consistency group on the backup storage device.
 12. A systemfor copying stored data and having the ability to recover from a failurecomprising: means for detecting a failure associated with a localstorage site having a local storage controller and a local storagedevice; means for re-synchronizing the local storage device after thefailure is corrected; and a remote storage controller having: means forasynchronously receiving updates from the local storage controller;means for writing the updates to a remote storage device means forstoring a consistent point in time copy of the updates on a backupstorage device; means for determining whether a group of pending updatesfor storage on the backup storage device form an intact consistencygroup, upon detection of the failure associated with the local site. 13.The system for copying stored data of claim 12 wherein the remotestorage controller further comprises means for taking corrective actionon the group of pending updates to create an intact consistency group.14. The system for copying stored data of claim 13 wherein thecorrective action is selected from a group of actions consisting ofcompleting a pending consistency group, withdrawing the pendingconsistency group, reverting to a prior intact consistency group andallowing the pending consistency group to become the intact consistencygroup.
 15. The system for copying stored data of claim 14 furthercomprising a recovery consistency group physically stored on the backupstorage device, formed from the intact consistency group.
 16. The systemfor copying stored data of claim 15 further comprising a remote host.17. The system for copying stored data of claim 16 wherein the remotestorage controller receives post-failure data updates from the remotehost.
 18. The system for copying stored data of claim 17 wherein themeans for re-synchronizing the local storage device after the failure iscorrected comprises the asynchronous copying of post-failure updates andupdates in the recovery consistency group from the remote storagecontroller to the local storage controller.
 19. The system for copyingstored data of claim 17 wherein the remote storage controller furthercomprises means for merging post-failure updates written to the remotestorage device with the recovery consistency group on the backup storagedevice, creating a new consistency group.
 20. An article of manufacturefor use in programming a data storage system to recover from a failure,the data storage system having a host computer writing updates to alocal storage controller associated with a local storage device, at alocal site having a local storage device, the local storage controllerasynchronously copying the updates to a remote storage controller at aremote site the remote storage controller storing the updates on aremote storage device and periodically storing a consistent point intime copy of the updates on a backup storage device, the article ofmanufacture comprising a storage medium having logic embedded therein tocause components of the data storage system to: detect a failureassociated with the local site; determine whether a consistent point intime copy of the updates pending for storage on the backup storagedevice at the time the failure is detected form an intact consistencygroup; correct the failure associated with the local site; andre-synchronize the local storage device.
 21. The article of manufactureof claim 20 wherein the logic further causes components of the datastorage system to take the following steps upon detecting the failureassociated with the local site: terminate the asynchronous copying ofupdates from the local storage controller to the remote storagecontroller; establish a reverse asynchronous update copying relationshipfrom the remote storage controller to the local storage controller; andquiesce the host computer.
 22. The article of manufacture of claim 20wherein the logic further causes components of the data storage systemto take corrective action on the updates pending for storage on thebackup storage device to create an intact consistency group.
 23. Thearticle of manufacture of claim 22 wherein the corrective action takenis selected from a group of actions consisting of completing a pendingconsistency group, withdrawing the pending consistency group, revertingto a prior intact consistency group and allowing the pending consistencygroup to become the intact consistency group.
 24. The article ofmanufacture of claim 23 wherein the logic further causes components ofthe data storage system to physically copy to the backup storage devicethe updates in the intact consistency group, thereby forming a recoveryconsistency group.
 25. The article of manufacture of claim 24 whereinthe logic further causes components of the data storage system tosynchronize the updates stored on the remote storage device with theupdates of the recovery consistency group.
 26. The article ofmanufacture of claim 25 wherein the logic further causes components ofthe data storage system to write post-failure updates directly to theremote storage controller.
 27. The article of manufacture of claim 26wherein the logic further causes components of the data storage systemto write the post-failure updates to the remote storage controller froma remote host.
 28. The article of manufacture of claim 26 wherein thelogic further causes components of the data storage system tore-synchronize the local storage device by synchronizing the localstorage device with the recovery consistency group and the post-failureupdates through the asynchronous copying of the updates from therecovery storage controller to the local recovery controller.
 29. Thearticle of manufacture of claim 26 wherein the logic further causescomponents of the data storage system to take the following steps afterthe local storage device is re-synchronized: quiesce the recovery host;terminate the asynchronous copying of updates from the remote storagecontroller to the local storage controller; re-establish theasynchronous copy relationship from the local storage controller to theremote storage controller; and store a new consistent copy of the dataon the remote storage device to the backup storage device.
 30. Thearticle of manufacture of claim 29 wherein the logic further causescomponents of the data storage system to form a new consistent copy ofthe data stored on the remote storage device on the backup storagedevice by merging post-failure updates written to the remote storagevolume with the recovery consistency group on the backup storage device.